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Requirement: 

To  develop  a  methodology  and  user's  guide  for  correcting  deficiencies 
in  Air  Defense  training  packages  that  are  discovered  as  a  result  of  training 
effectiveness  evaluations  (TEEs). 

Procedure: 

The  guidelines  for  revising  training  program  deficiencies  were 
designed  in  order  to  provide  training  analysts  with  solutions  for  correcting 
problems  in  training  identified  during  a  TEE.  Using  prior  developmental  work 
by  the  Army  Research  Institute  Field  Unit  Fort  Knox  in  the  area  of  Training 
Program  Evaluation  as  the  baseline,  this  effort  involved  three  steps: 

(1)  Identify  discrepancies  in  training  program  processes  and 
products  on  the  basis  of  ratings  to  the  questions  included  on  the  TEE; 

(2)  Trace  discrepancies  to  their  underlying  causes;  and 

(3)  Recommend  specific  modifications  to  processes  and  products 
which  would  increase  training  effectiveness. 

The  issues  addressed  by  the  Revision  Guidelines  directly  correspond 
to  the  training  program  components  incorporated  in  the  TEE.  The  TEE  methodology 
provides  a  mechanism  for  analyzing  training  processes  (e.g.,  how  training  is 
conducted),  which  was  the  focus  of  the  earlier  Harless  Guidelines,  and  also 
for  evaluating  training  products  (e.g.,  the  design  of  instruction),  which  had 
not  been  a  part  of  training  evaluation  methodology.  The  Revision  Guidelines 
provide  specific  solutions  for  correcting  problems  in  these  areas. 

In  format,  the  guidelines  follow  the  general  scheme  used  by  the 
job  aid  for  modifying  training  programs (Research  Product  81-17;  Kristiansen, 
1981).  A  total  of  29  problem  headings  were  categorized  under  four  subdomains 
of  training,  which  were  (1)  Training  Environment,  (2)  Presentation,  (3)  Prac¬ 
tice/Demonstration,  and  (4)  Testing.  Each  problem  was  defined,  related  to 
specific  TEE  worksheets  and  questions,  and  discussed  in  terms  of  the  specific 
actions  needed  to  correct  it. 


Findings: 


The  TEE  methodology  and  companion  revision  guidelines  represent  a 
broader  and  more  complete  evaluation  methodology  than  was  previously  available 
for  identifying  and  correcting  shortcomings  in  Army  Air  Defense  packages. 

The  TEE  system  is  fully  documented  in  the  TEE  User's  Guide,  and  was  shown  to 
be  applicable  to  Air  Defense  training  processes  and  products  during  a  pre¬ 
liminary  validation.  The  Revision  Guidelines,  while  they  have  not  been  tested 
empirically,  correspond  closely  to  the  TEE  in  subject  content  matter. 

Utilization  of  Findings: 

The  methodologies  developed  for  evaluating  and  revising  training 
program  components  can  potentially  contribute  to  the  more  effective  use  of 
training  by  specifying  deficient  areas  and  the  means  for  correcting  discrepan¬ 
cies.  When  applied  to  ongoing  Air  Defense  training  packages  or  training  for 
operational  tests  (OTs),  they  provide  an  important  input  to  design  considerations 
and  the  conduct  of  training. 
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Section  I 


INTRODUCTION 


Background 

Currently  the  Army  is  engaged  in  the  development  and/or  fielding  of 
four  new  Air  Defense  systems  -  STINGER,  ROLAND,  SGT  YORK,  and  PATRIOT.  The 
design  and  implementation  of  these  systems  offers  great  potential  for  increasing 
the  Army's  capability  to  provide  effective  Air  Defense.  Recognizing  that 
the  new  weapons  will  only  be  as  effective  as  the  soldiers  who  man  them, 
however,  the  Army  requires  that  programs  and  material  to  train  or  retrain 
soldiers  to  these  system  be  implemented  concurrently  with  the  fielding  of 
each  system.  TRADOC  Reg  350-7  {Not  referenced]  provides  Army  policy  governing 
a  systems  approach  to  training.  It  prescribes  in  generic  terms  the  minimum 
requirements  of  what  must  be  done  to  insure  the  systematic  evaluation,  analysis, 
design,  development,  and  implementation  of  training  programs  and  training 
support  materials.  The  TRADOC  model  for  a  systems  approach  to  training 
incorporates  the  same  concept  as  the  Interservice  Procedures  for  Instructional 
Systems  Development  model,  commonly  known  as  the  ISD  model,  but  is  purposely 
expressed  in  a  non-linear  fashion  to  demonstrate  the  continuing  nature  of  a 
systems  approach  and  the  interdependence  of  its  process.  It  also  allows  for 
entry  at  any  point  as  the  result  of  evaluation. 

The  Air  Defense  community  has  expressed  the  need  for  methodologies 
to  evaluate  the  training  programs  for  these  systems  during  the  operational 
test  (OT)  phase  of  the  developmental  cycle.  These  needs  focused  on  three 
specific  concerns. 

1.  Methodologies  for  conducting  training  effectiveness  evaluations 
(TEE)  of  Air  Defense  training  packages  during  the  OT  phase; 

2.  Materials  and  Methods  for  performing  training  effectiveness 
testing  (TET)  of  Air  Defense  training  devices  during  the  validation,  verifi¬ 
cation  and  system  integration  phases;  and 

3.  Methodologies  for  applying  feedback  information  from  TEE's  and 
TET  to  improve  training  packages  and  devices. 

The  Army  Research  Institute  has  undertaken  to  address  each  of  these 
needs  through  support  of  research  in  the  development  of  materials,  methodologies 
and  procedures  for  conducting  TEEs  and  TETs.  Specifically,  the  development 
of  a  TEE  methodology  was  addressed  by  research  performed  by  Calspan  Corporation’s 
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Advanced  Technology  Center  under  the  technical  direction  of  ARI's  Field  Unit 
at  Fort  Bliss  (ARIFUFB).  This  effort,  performed  under  Contract  Number 
MDA903-80-C-0579  during  the  period  August  1980  -  October  1981,  is  documented 


in  the  Year  1  final  report  Development  of  Guidelines  for  Conducting  Training 
Effectiveness  Evaluation  of  Air  Defense  Training  (Fishbume,  Rolnick,  &  Larsen, 
1981) . 


The  first  year's  effort  entailed  development  of  methodologies  for 
evaluating  training  packages  during  OT  that  was  consistent  with  Instructional 
Systems  Development  (ISD)  procedures  and  designing  procedures  for  the  struc¬ 
tural  observation  and  assessment  of  the  training  process.  The  development 
of  the  TEE  and  resultant  outputs  (e.g. ,  TEE  materials  and  User's  Guide)  are 
discussed  below  in  greater  detail. 

The  second  year  of  the  research  focused  on  the  third  need  identified 
by  the  Air  Defense  community  noted  above,  namely,  the  development  of  a  method¬ 
ology  for  applying  the  results  of  a  TEE  in  the  improvement  of  training  programs 
In  addition,  the  user's  guidelines  for  conducting  TEE's  were  revised  and  refined 
The  efforts  and  products  of  Calspan's  research  are  documented  in  this  report. 

Purpose  and  Scope 

The  overall  objectives  of  the  research  were  (1)  to  update  and  revise 
existing  methodologies  for  evaluating  Army  Air  Defense  packages  and  (2)  to 
develop  a  set  of  guidelines  for  revising  training  discrepancies  identified  by 
a  TEE.  The  purpose  of  the  first  year's  research  was  to  validate  and  refine 
the  Harless  Guidelines  (1980)  for  training  program  evaluation.  As  reported  in 
the  first  year's  report,  the  researchers  conducted  a  preliminary  validation  of 
the  Harless  Guidelines.  Based  on  the  findings,  the  TEE  Guidelines  were 
developed  to  broaden  the  earlier  evaluation  job  aid  by  incorporating  product 
and  courseware  evaluation. 

The  purpose  of  the  second  year  research  was  to  extend  the  TEE 
methodology  to  Include  the  revision  process.  The  primary  assumption  under¬ 
lying  the  effort  was,  simply,  that  evaluation  is  pointless  unless  action  is 


taken  to  correct  the  deficiencies  and  discrepancies  revealed  by  that  process. 
Based  on  earlier  work  in  training  program  revisions,  notably,  the  Seville 
methodology  (Spears,  Maxey,  and  Roush,  1980)  and  Kristiansen's  (1981)  job  aid 
for  modifying  training,  the  approach  was  taken  to  trace  symptoms  of  discrepancies 
in  training  products  and  procedures  directly  to  their  causes  and  to  specify 
detailed  recommendations  for  modifying  them.  The  effort  is  broader  than 
earlier  methodologies  with  respect  to  the  detail  of  specific  solutions  and 
the  scope  of  deficiencies  addressed,  especially  in  the  areas  of  product  and 
courseware  revision.  The  major  deliverable  from  this  effort  is  a  set  of 
guidelines  for  applying  feedback  information  from  a  TEE  in  order  to  improve 
training  packages. 


Section  II 


DEVELOPMENT  OF  GUIDELINES  FOR  TRAINING  PROGRAM  REVISIONS 

A  number  of  sources  were  used  in  the  development  of  the  Guidelines 
for  Revising  Training  Program  Deficiencies .  Since  the  revision  guidelines 
were  designed  as  a  companion  manual  to  guidelines  for  conducting  TEEs  for 
TEE  analysts,  to  be  initiated  in  response  to  a  TEE,  the  detailed  problems 
and  associated  solutions  directly  correspond  to  the  domain  covered  by  the 
Guidelines  for  conducting  a  TEE. 

Many  of  the  documents  referenced  in  TEE  development  were  useful 
in  both  the  conceptual  and  technical  aspects  of  the  developmental  process 
during  the  TEE  development  phase.  These  were  used  to  provide  standards  for 
rating  the  adequacy  of  training  products  and  procedures.  During  the  develop¬ 
ment  of  the  Revision  Guidelines,  these  sources  provided  the  basic  material 
for  specifying  the  solutions. 

In  addition,  the  issues  addressed  by  the  Seville  Revision  Methodology 
(Spears,  Maxey  &  Roush,  1980),  Kristiansen’s  (1980)  working  paper  in  training 
program  evaluation  (TPE)  modification,  and  especially  the  content  and  structure 
of  Kristiansen's  (1981)  job  aid  for  modifying  ineffective  or  inefficient 
training  programs,  directly  influenced  the  final  form  of  the  guidelines. 


The  TEE  Methodology 

As  background  for  the  development  of  the  Revision  Methodology, 
it  is  important  to  understand  the  basic  components  and  procedures  involved 
in  a  TEE.  A  complete  description  and  rationale  for  TEE  is  contained  in 
the  final  report  for  the  first  year  (1981). 

The  design  of  TEE  methodology  involved  the  identification  of  the 
purposes  and  scope  of  evaluating  training  effectiveness,  and  the  development 
of  a  methodology  and  a  set  of  job  aids  for  conducting  the  TEE.  The  Calspan 
effort  focused  on  building  an  assessment  tool  on  the  baseline  established 
by  the  Harless  Guidelines  (1980).  Examination  of  the  Guidelines  indicated 
that  they  were  adequate  with  respect  to  the  process  evaluation  component,  the 
conduct  of  training,  but  did  not  address  several  components  considered 
important  in  a  broader  approach  to  training  evaluation.  Foremost  was  the 
exclusion  of  a  product  evaluation  methodology,  involving  an  analysis  of 
training  design  based  on  training  documentation.  Therefore,  the  Instructional 
Quality  Inventory  (IQI)  (Wulfeck  et  al.,  1978;  Ellis  et  al.,  1978,  1979; 
Fredericks,  1980)  was  used  as  a  basis  for  developing  the  TEE  product  evaluation 
component. 

Development  of  the  TEE  procedures  also  followed  the  general  struc¬ 
ture  of  the  Harless  Guidelines,  but  with  several  important  differences.  A 
master  list  of  94  evaluation  questions  was  developed  to  identify  performance 
discrepancies  and  to  enable  comparison  of  training  products  to  specified 
standards.  The  major  advantage  of  the  list  of  questions  is  its  flexibility 
for  allowing  a  TEE  analyst  to  select  applicable  areas  for  evaluation.  The 
procedure  for  identifying  discrepancies  by  comparing  training  components  to 
specified  standards  also  facilitates  the  revision  process  by  providing  a 
direct  approach  for  tracing  performance  discrepancies  to  their  probable  causes. 
The  Appendix  of  this  report  contains  the  short  form  of  the  Master  Questions. 


Other  procedural  considerations  involved  design  of  a  systematic 
method  for  carrying  out  a  TEE  from  its  initial  request  and  planning  phase 
to  the  documentation  of  the  TEE  effort  and  preparation  of  the  final  report. 


TEE  training  materials  and  job  aids  were  designed  to  assist  the  TEE  analyst 
and  other  data  collectors.  A  user's  manual  for  the  conduct  of  a  TEE  (1981) 
and  a  separate  data  collector's  handbook  (1981)  for  assisting  personnel  were 
developed  to  facilitate  the  TEE  process. 

Baseline  for  the  Revision  Guidelines 

For  the  most  part,  the  conceptual  and  procedural  approach  for 
designing  the  Guidelines  for  Revising  Training  Program  Deficiencies  was 
defined  in  the  process  of  developing  the  TEE  system.  Relevant  issues, 
decision  points,  and  coverage  followed  directly  from  the  TEE  itself,  and 
indirectly  from  the  materials  used  to  design  the  TEE.  However,  as  stated 
earlier,  three  sources  contributed  substantially  to  the  concepts  and  procedures 
related  to  the  Revision  Methodology.  These  are  (1)  the  Seville  Project  Report 
80-8  Methodology  for  Correcting  Deficiencies  in  Training  Programs  (Spears, 

Maxey  &  Roush,  1980);  (2)  Kristiansen's  (1980)  Training  Program  Evaluation. 

A  Job  Aid  for  Modifying  Ineffective  or  Inefficient  Training  (Working  Paper 
FKFU  80-8);  (3)  Kristiansen's  (1981)  A  Job  Aid  for  Modifying  Ineffective  or 
Inefficient  Training  Programs  (Research  Product  81-17). 

The  Seville  project  sought  to  extend  the  Harless  Guidelines  by 
developing  procedures  for  correcting  training  deficiencies  discovered  in  a 
TPE.  Their  major  objective  was  to  provide  a  set  of  guidelines  or  principles 
which  could  be  used  by  training  evaluators  to  direct  the  revision  process. 
Although  the  focus  was  on  the  XM1  tank  (as  was  the  Harless  Guidelines'),  the 
researchers  sought  to  make  the  guidance  general  enough  to  have  applicability 
to  any  training  program  where  skilled  performance  was  involved.  The  scope 
of  the  project  was  to  specify  principles  for  correcting  (a)  deficiencies 
identified  by  the  Harless  Guidelines  and  traceable  to  specifiable  instructional 
sources  through  systematic  observation;  and  (b)  those  attributable  to 
administration  of  training  described  in  the  Harless  Guidelines  as  "non-skill 
related." 


The  methodology  focused  on  systematic  observation  and  analysis  of 
instructional  processes  and  their  effects  on  the  daily  achievements  of  trainees 
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Practice  -  the  amount,  type,  and  schedule  -  was  postulated  to  be  the  under¬ 
lying  determinant  of  skill  development  and  to  provide  opportunity  for  the 
constructive  application  of  guidance  and  feedback.  Fifteen  principles  concerned 
with  practice,  guidance  and  feedback,  provide  the  conceptual  framework  for 
describing  a  good  training  program.  These  principles  are  as  follows: 

1.  A  person  learns  to  do  whatever  he  practices  in  the  learning 

situation. 

2.  The  total  amount  of  practice  should  be  sufficient  to  ensure 
stable  criterion  performance. 

3.  The  length  and  frequency  of  practice  sessions  should  be 
adapted  to  the  task  being  Caught,  the  experience  of  the  learner,  and  the 
opportunities  for  performing  the  task  in  the  future. 

4.  Progress  should  be  made  during  practice. 

5.  The  learner  should  be  intensively  involved  during  practice. 

6.  Guidance  is  needed  when  the  learner  cannot  determine  a  correct 
action  on  his  own. 

7.  Guidance  should  direct  the  learner's  attention  to  individual 
cues  and  specific  performance  requirements. 

8.  Verbal  knowledge  from  classroom  instruction,  text  material 
memory  aids,  and  job  aids  should  be  presented  and  their  use  practiced  in 
ways  that  capitalize  on  their  guiding  value. 

9.  Guidance  should  not  be  used  when  it  is  not  needed. 

10.  Self-guidance  should  be  a  specific  goal  in  all  training. 

11.  Feedback  should  be  specific  to  the  response. 

12.  Feedback  should  occur  while  the  learner  still  remembers  clearly 
what  he  did. 

13.  When  the  ongoing  coordination  of  an  action  depends  on  con¬ 
tinuous  feedback  from  each  part  of  the  action,  no  delay  in  feedback  can  be 
tolerated. 

14.  Feedback  should  be  provided  in  ways  that  help  the  learner  see 
when  he  is  making  progress. 

15.  When  errors  are  made,  or  when  there  is  little  or  no  progress  in 
skill  acquisition,  the  feedback  should  focus  on  correcting  the  difficulties, 
not  punishing  the  learner. 
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Instructional  deficiencies  are  evidenced  through  the  systematic 
observation  of  the  training  process  from  the  Harless  Guidelines.  These 
guidelines  list  more  than  twenty  specific  deficiencies  in  the  Seville  Report. 
These  deficiencies  are  characterized  by  five  aspects  of  trainer  behavior 
which  signal  problems  in  the  instructional  system.  These  "symptoms"  of 
instructional  discrepancies  are: 

1.  No  hands-on  practice  is  provided. 

2.  Trainees  do  not  know  what  to  do  in  the  presence  of 
relevant  cues. 

3.  Performance  shows  undue  variability  from  one  trial  to 
another. 

4.  Trainees  are  unduly  slow  in  performing  a  task. 

5.  There  is  too  much  dependence  on  job  or  memory  aids. 

Evidence  of  the  presence  for  one  or  more  of  these  symptoms  suggests 
that  the  particular  component  of  training  under  observation  fails  to  adhere 
to  one  or  more  of  the  training  principles  listed  above,  and  that  the  procedures 
need  to  be  modified  in  line  with  the  underlying  principle. 

The  Seville  Report  also  addresses  deficiencies  in  training  programs 
that  are  determined  primarily  by  the  training  system  design  or  administrative 
management  practices.  Five  categories  of  shortcomings  are  reviewed:  (1)  Indi¬ 

cations  of  Need  to  Redesign  Training  Programs;  (2)  Training  Support  and 
Facilitation;  (3)  Instructor  Characteristics  and  Qualifications;  (4)  Manage¬ 
ment  of  Attitudes;  and  (5)  Problems  in  Performance  Testing. 


Indications  of  Need  to  Redesign  Training  Programs  discusses  15 


problems  in  training  outcomes  identified  by  the  Harless  report  that  involve 
general  shortcomings  in  the  design  of  a  program.  For  example,  the  deficiency 
"Pertinent  tasks  are  omitted  from  training"  suggests  that  task  analysis 
procedures  were  not  initially  followed,  and  therefore,  imply  that  general 


revisions  of  the  program  are  needed  from  top  to  bottom,  requiring  substantial 
administrative  support  and  resources. 


•  Training  Support  and  Facilitation,  covers  12  factors  which  signal 
an  administrative  neglect  of  training  program  quality,  for  example,  through 
adverse  ratios  of  trainees  to  resources  (instructors,  equipment,  space),  lack 
of  facility  communication  or  quality  assurance  practices,  etc.  Correction 

of  problems  requires  administrative  attention  and  resolve,  and  some  may  in 
addition  necessitate  additional  resources. 

•  Instructor  Characteristics  and  Qualifications  addresses  three 
problems,  namely  skill  of  the  content  area,  skill  in  teaching,  and  attitude. 
Judicious  use  of  human  resource  management  principles  can  resolve  specific 
problems. 

•  Management  of  Attitudes  lists  four  problems  involving  negative 
attitudes  or  their  underlying  causes.  Understanding  attitude  formation  and 
incorporating  positive  motivational  techniques  and  reward  systems  are  dis¬ 
cussed  as  solutions. 

•  Problems  in  Performance  Testing  deals  with  16  Harless-identified 
problems  in  the  testing  component.  Representative  examples  are  "Mismatch 
between  training  and  test  conditions"  and  "Poor  test  administration."  Careful 
test  design  and  standardization  procedures,  and  consideration  of  the  systematic 
relations  between  training  and  testing  are  called  for. 

The  Seville  report  provided  an  approach  to  training  program  modi¬ 
fication  which  has  widely  influenced  later  researchers  in  the  area.  Either 
implicitly  or  explicitly,  the  principles  and  specific  solutions  documented 
in  the  methodology  provided  a  conceptual  baseline  for  guiding  thought  on 
correcting  deficient  training  processes.  The  report  emphasized  practice  as 
the  critical  component  of  training,  reliance  on  identifying  and  correcting 
underlying  causes  of  performance  discrepancies,  and  most  importantly,  support 
of  the  systematic  approach  to  training  development.  Its  principal  limitations, 
addressed  by  later  research,  include  (1)  lack  of  a  clearcut  methodology  for 
the  evaluator  to  directly  bridge  between  the  discovered  deficiency  and  its 
correction,  and  (2)  relative  neglect  of  courseware  deficiencies. 

Kristiansen  (1980)  addressed  the  first  of  these  constraints  by 
developing  a  job  aid  for  correcting  training  problem  deficiencies  which 
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listed  35  problems  corresponding  to  specific  documented  deficiencies  from  a 
TPE;  each  problem  was  associated  directly  with  one  or  more  modifications 
to  bring  the  training  program  in  line  with  ISO  specifications.  This  working 
paper  served  as  the  prototype  in  terms  of  its  problem-solution  format  for 
his  1981  job  aid  and  for  the  present  guidelines. 


ARI  as  the  baseline  for  the  present  project.  The  job  aid  is  one  of  four  to 
perform  training  program  evaluations;  its  specific  purpose  is  to  assist 
training  analysts  in  identifying  solutions  for  problems  in  training  processes 
discovered  during  a  TPE. 

Although  Kristiansen  (1981)  follows  the  same  general  methodology  as 
the  earlier  working  paper,  the  scope  of  problems  and  solutions  was  greatly 
expanded  in  comparison  to  the  working  paper.  Underlying  the  job  aid  is  a 
generic  approach  to  evaluation  which  assumes  that  training  programs  should 
incorporate  a  sequence  of  five  activities: 

1.  Enabling  knowledge 

2.  Demonstration 

3.  Part- Task  Practice 

4.  Whole- Task  Practice 

5 .  Tes  ting 

The  heart  of  the  job  aid  is  21  problem  areas  which  could  potentially 
be  encountered  in  training  environments.  Each  problem  describes  one  or  more 
specific  deficiencies  identified  by  items  from  the  Job  Aid  for  the  Structured 
Observation  of  Training  (Research  Product  81-16).  In  addition,  specific  fixes 
for  the  discrepancies  are  identified.  Further,  the  problems  are  grouped 
under  five  major  headings  of  training.  The  five  categories  and  the  problems 
discussed  are  as  follows: 

1.  Training  Environment 

a.  Number  of  Instructors 


b.  Training  Equipment 

c.  Training  Site 

d.  Distractions 

e.  Training  Duration 

2.  Lecture  Events 

a.  Training  objectives  and  purpose 

b .  Terminology 

3.  Demonstration/Practice 

a.  Demonstration 

b.  Practice 

c.  Feedback 

d.  Job  Aids 

e.  Progression  of  Training  Events 

4.  General  Observations 

a.  Implementation  of  Training  Plan 

b.  Training  Aids 

c.  Instructor  Performance 

d.  Training  Duration 

5.  Testing 

a.  Test  Instructions 

b.  Test-training  Match 

c.  Test  Realism 

d.  Contamination 

e.  Feedback 

The  Kristiansen  (1981)  job  aid  provided  a  strong  basis  for  the 
present  set  of  revision  guidelines  and  much  of  the  material  has  been  incor¬ 
porated  in  the  present  handbook.  However,  the  job  aid  dealt  explicitly  only 
with  training  processes  and  needed  to  be  extended  to  both  the  design  of 
training  and  the  quality  of  instructional  courseware,  following  the  efforts 
of  TEE  development.  In  addition,  team  and  collective  training  was  not  a 
major  consideration  in  this  job  aid.  Although  the  guidelines  are  for  correcting 
individual  components,  they  are  also  relevant  to  collective  training  components. 
Specific  consideration  of  team  practice  needs  to  be  addressed.  Further 
discussion  of  these  points  is  contained  in  the  first  year's  report. 


Section  III 


DESCRIPTION  OF  THE  REVISION  GUIDELINES 

Problems  and  solutions  are  identified  in  the  Guidelines  for 
Revising  Training  Deficiencies  by  categorizing  them  under  four  broad  training 
system  components.  This  organization  is  largely  similar  to  that  used  by 
Kristiansen  C1981)  and  generally  represents  distinguishable,  interdependent 
domains  for  assessing  training.  The  broad  topical  areas  are:  (1)  Training 
Environment;  (2)  Presentation;  (3)  Practice/Demonstration;  and  (4)  Testing 
Kristiansen  also  used  a  "General  Observations"  category;  problems  clustered 
under  that  heading  which  were  incorporated  into  the  TEE  were  reclassified 
under  one  of  the  four  categories  used  here. 

Within  each  of  the  broad  headings,  specific  problems  and  their 
solutions  are  detailed  in  three  steps  that  follow  the  methodology  of 
Kristiansen: 

Problem:  the  nature  of  the  deficiency  is  defined  in  terms  of  the 
observable  discrepancies  of  the  training  event. 

Evidence:  sources  for  problem  identification  are  listed.  These 
include  the  TEE  worksheets  which  were  used  to  document  the  deficiency 
and  the  specific  TEE  items  related  to  the  defined  problem. 

Recommendation  for  modifying  deficiency.  For  each  of  the  specific 
elements  (TEE  items) ,  standards  of  adequate  conditions  and  ways  of  achieving 
standards  through  program  modification  are  addressed. 

Many  of  the  solutions  are  straightforward  fixes  (e.g.,  if  the 
environment  is  too  noisy,  the  solution  involves  reducing  the  noise  level 
or  taking  other  steps  to  insure  that  the  noise  does  not  interfere  with 
accomplishing  the  objective).  However,  as  noted  by  Kristiansen  and  others 
(Spears,  et  al.  1980),  the  transparency  of  many  training  problems  and  their 
solutions  has  apparently  had  little  impact  on  making  appropriate  corrections. 
The  guidelines  are  organized  as  follows : 


1.  Training  Environment  covers  training  deficiencies  which  appear  to 

have  a  pervasive  influence  on  many  more  specific  aspects  of  the  training 
system.  Generally,  the  problems  grouped  under  this  category  are  symptomatic 
of  deficiencies  in  administrative  responsibility  (e.g.,  Spears  et  al.,  1980) 
and  may  require  more  global  administrative  intervention  to  be  modified. 

Problem  areas  (with  their  related  TEE  Questions)  include  the  following: 

a.  Training  site  (TEE  Questions  85,  86  also  under  lg,  87,  88)  addres¬ 
ses  environmental  conditions  including  space,  noise,  lighting  and  temperature. 

b.  Implementation  of  Training  Plan  (Questions  82,  83,  84)  details 
deficiencies  and  contamination  of  the  actual  content  of  training  vis-a-vis 
lesson  plans  and  instructor  guidelines. 

c.  Number  of  Instructors  (  Question  81)  considers  the  effect  of 
ins true tor /trainee  ratio  on  adequate  guidance  and  feedback  activities. 

d.  Training  duration  (Questions  90,  92)  deals  with  the  length  of 
time  devoted  to  the  training  course  and  with  rest  periods. 


e.  Attitudes  and  Motivational  Techniques  (Questions  27,  28,  89) 
address  potential  solutions  to  attitudinal  and  motivational  deficiencies. 

f.  Equipment  (Question  93)  relates  to  proper  functioning  of 
training  devices. 

g.  Distractions  (Question  86)  address  interruptions  in  training 
events  such  as  equipment  breakdowns,  instructor  absences »  and  visitors. 

2.  Presentation  encompasses  a  number  of  issues  that  are  related  to 

the  adequacy  of  the  plan  for  training,  the  lecture  format,  and  the  details  of 
getting  information  across  to  trainees.  Many  of  the  problems  addressed  in 
this  category  have  been  labelled  instructional  deficiencies  (Spears,  et  al., 
1980)  and  are  similar  to  those  grouped  in  Kristiansen's  Lecture  category. 

The  problems  under  Presentation  include  the  following: 

a.  Prerequisites  (Questions  29,  30)  address  incoming  skills  and 
knowledges  required  of  trainees. 

b.  Training  objectives  (Question  31)  considers  whether  the 
objectives  of  training  were  presented  to  the  student  at  the  beginning  of 
each  lesson. 


c.  Course  Administration  Directions  (Questions  79,80)  deals  with 
the  completeness  and  realistic  demands  of  conducting  lesson  plans. 

d.  Adequacy  (Questions  32,  33,  34,  35)  refers  to  the  completeness 
of  the  basic  presentation  components  presented  during  lectures  for  facts, 
concepts,  procedures  and  rules. 

e.  Sequencing  (Questions  75,  76)  deals  with  the  order  of  objec¬ 
tives  within  lesson  plans  and  also  the  order  of  lesson  plans  in  a  course. 

f.  Clarity  and  comprehensibility  (Questions  69,  70,  71,  72,  91,  94) 
considers  the  problems  and  solutions  related  to  spoken  and  written  instruc¬ 
tional  materials,  including  the  identification,  technical  quality,  and  ease 

of  understanding. 

g.  Examples  (Questions  37,  38,  39,  40,  41)  addresses  the  deficiencies 
and  corrections  for  examples  and  includes  their  adequacy,  sequence,  and  the 

use  of  non-examples. 

h.  Media  (Questions  77,  78)  covers  the  appropriateness  and 
adequacy  of  training  devices  such  as  equipment,  slides,  and  text  for  conveying 
the  information  in  lesson  plans. 

i.  Visuals  (Questions  73,  74)  refers  to  the  use  and  adequacy 
of  pictures  in  enhancing  instruction. 

3.  Practice/Demonstration  includes  eight  potential  problems  which 

could  adversely  affect  training  effectiveness.  These  problems  address  per¬ 
formance  oriented  activities  requiring  hands-on  practice  and  associated 
elements.  Since  soldiers  will  not  learn  to  perform  the  tasks  required  in 
operational  settings  without  hands-on  practice,  the  identification  of 
problems  in  this  category  and  their  successful  resolution  is  essential  to 
optimal  training  outcomes.  The  following  problems  are  defined  and  addressed: 

a.  Practice  (General) (Questions  58,  59,  60,  61,  62)  refers  to 
motor  aspects  of  practice,  including  objective-practice  consistency  and  com¬ 
pleteness,  evidence  that  trainees  practice,  and  final  practice. 

b.  Practice  Remembering  (Questions  45,  46)  details  the  use  of 


memory  aids  and  practice-test  consistency. 


c.  Practice  Using  (Questions  47,  48,  49,  50)  denotes  problems 
related  to  USE  or  performance  training,  including  their  sequence,  adequacy, 
and  integration. 


d.  Demonstrations  (Questions  42,  43,  44)  addresses  adequacy  and 
effectiveness  of  instructive  demonstrations. 

e.  Team  Practice  (Questions  65,  66,  67,  68)  encompasses  potential 
shortcomings  and  modifications  for  training  team  functions,  including  its 
provision,  realism,  and  adequacy  of  feedback. 

f.  Feedback  (Questions  63,  64)  deals  with  the  knowledge  of  results 
provided  to  trainees  during  practice. 

g.  Job  Performance  Aids  (Questions  36,  51,  52)  considers  the  use, 
adequacy  and  explanation  of  training  aids. 

h.  Practice-test  Consistency  (Questions  53,  54,  55,  56,  57) 
examines  the  relationship  of  items  practiced  to  those  tested,  including  level, 
type,  format,  condition,  and  standards. 

4.  Testing,  addresses  the  problems  and  recommended  methods  for  increasing 

effectiveness  of  tests.  The  deficiencies  covered  under  this  category  include 
those  related  to  the  instructional  design  of  testing  materials  and  the  way 
in  which  testing  is  carried  out.  Five  generic  problem  areas  are  discussed 
along  with  recommended  solutions.  These  are: 

a.  Objectives- test  consistency  (Questions  1,  2,  3,  4,  5,  6,  7) 
covers  the  match  between  learning  objectives  and  corresponding  test  questions. 
The  consistency  of  test  item  format,  task  level,  concent,  conditions,  and 
standards  to  learning  objectives  is  discussed. 

b.  Test  instructions  (Questions  21,  22,  23)  deals  with  the  com¬ 
pleteness  and  clarity  of  test  directions  and  the  degree  to  which  instructors 
adhered  to  them  during  test  administration. 

c.  Test  adequacy  (Questions  8,  9,  11,  12,  13,  14,  15,  16,  17,  18, 

19,  20)  involves  the  design  of  test  items,  answer  keys,  and  test  formats  and 
recommendations  for  modifications. 

d.  Test  realism  (Questions  10,  24,  25)  considers  the  relationship 
of  the  test  to  the  conditions  and  requirements  of  the  job  being  trained. 

e.  Contamination  (Question  26)  addresses  the  amount  of  help  and 
external  cues  available  during  testing  which  may  invalidate  test  results. 


Section  IV 


SUMMARY  AND  CONCLUSIONS 

Training  Effectiveness  Evaluation  (TEE)  offers  one  means  of  fulfilling 
many  evaluation  requirements  within  the  context  of  a  systems  approach  to 
training.  The  scope  of  a  TEE  may  vary  according  to  management  needs  and 
available  resources,  but  the  main  purpose  is  always  to  identify  training- 
related  problems  and  to  suggest  underlying  causes.  A  TEE  may  be  conducted  in 
support  of  an  operational  test  of  a  new  system,  or  it  may  address  on-going 
training  in  a  review  capacity  or  in  response  to  complaints.  Within  a  specific 
TEE,  data  collection  may  encompass  product  evaluation,  process  evaluation,  or 
both.  Product  evaluation  involves  an  analysis  of  training  design  based  on 
training  documentation.  Process  evaluation  examines  the  conduct  of  training. 

If  the  TEE  addresses  both  product  and  process  issues,  performance  discrepan¬ 
cies  can  be  identified  and  corresponding  training  deficiencies  can  be  isolated 
from  the  perspective  of  both  design  and  implementation.  In  this  case,  much 
of  the  diagnostic  work  necessary  to  specify  revision  requirements  is  accom¬ 
plished  for  the  training  manager  and  ISD  support  personnel.  If  only  a  product 
evaluation  is  conducted,  only  design  deficiencies  can  be  identified.  This 
can  be  useful,  however,  as  a  quality  control  mechanism  prior  to  investigating 
new  discrepancies  and  deficiencies  in  the  conduct  of  training,  provided  that 
the  adequacy  of  objectives  and  the  adequacy  and  consistency  (with  objectives) 
of  the  tests  can  be  verified.  Thus,  a  product  evaluation  component  is  a 
necessary  component  of  a  process  evaluation. 

Evaluation  is  not  to  be  considered  simply  as  the  process  following 
implementation.  The  term  "evaluate"  is  used  in  the  general  judgmental  sense 
of  the  continuous  monitoring  of  a  program  or  of  the  training  function  as  a 
whole  and  involves  both  verification  and  validation.  The  process  consists  of 
internally  evaluating  the  training  program  during  each  phase  of  its  preparation 
(to  the  degree  that  resources  permit)  while  concurrently  externally  evaluating 
the  overall  training  function.  Thus,  following  implementation,  feedback  is 
used  to  evaluate  the  program,  assess  the  quality  of  soldiers'  performances 
and  check  the  organization’s  responsiveness  to  training  needs.  If  evaluation 
of  the  program  indicates  a  need  for  change,  the  development  cycle  is  reentered 
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at  the  appropriate  point.  In  a  broader  sense  evaluation  is  a  continual, 
empirically  based  process  that  involves  the  verification,  validation,  and 
assessment  of  data  used  by  the  training  system,  to  insure  that  the  needs  of  the 
Army  are  met  in  producing  trained  soldiers  and  combat  ready  units.  It  is  im¬ 
portant,  therefore,  to  obtain  a  variety  of  complementary  data  from  different 
sources;  evaluation  based  on  the  evidence  of  only  one  type  of  feedback  may  be 
unreliable  and  lack  validity.  It  should  be  remembered  that  evaluation  is 
pointless  unless  action  is  taken  to  correct  deficiencies  and  discrepancies 
revealed  by  that  process,  and  chat  the  management  function  of  providing  for 
quality  control  is  equally  important  during  each  phase  of  the  training  process. 


The  Guidelines  for  Revising  Training  Proera 


described 


in  this  report  provides,  we  believe,  a  broader  and  more  complete  method  for 
revising  training  program  curriculum  and  procedural  discrepancies  than  has 
heretofore  been  available  to  Army  Air  Defense.  As  a  part  of  the  TEE  system, 
the  guidelines  provide  the  specific  means  for  increasing  training  effectiveness 
by  correcting  the  major  discrepancies  revealed  by  the  assessment.  Together 
with  the  TEE,  the  guidelines  provide  extension  of  the  evaluation  methodology 
initiated  by  the  Harless  Guidelines  and  developed  further  by  Kristiansen  and 
his  coworkers  at  the  Army  Research  Institute.  The  present  results  and  documents 
are  an  additional  step  in  refining  instructional  systems  evaluation  technology 
to  increase  training  effectiveness. 
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1.  Is  the  TEST  FORMAT  appropriate 
for  the  OBJECTIVE? 

2.  Are  there  test  items  for  the  TLO 
or  all  of  its  critical  parts/LOs’ 
(See  job  aid  for  critical  parts.) 


3.  Is  there  a  test  item  for  each 
critical  part  of  each  LO? 

(See  job  aid  for  critical  parts.) 


4.  Does  the  TASK  LEVEL  of  the 
test  item  match  the  TASK  LEVEL 
of  its  OBJECTIVE? 

5.  Does  the  content  of  the  test  item 
match  the  content  of  its 
OBJECTIVE? 

6.  Do  the  CONDITIONS  of  the  test 
item  match  the  CONDITIONS 

of  its  OBJECTIVE? 

7.  Do  the  STANDARDS  of  the  test 
item  match  the  STANDARDS  of 
its  OBJECTIVE? 

8.  For  true-false,  multiple  choice, 
and  matching  items  is  only  one 
answer  correct? 

9.  For  short  answer,  fill-in,  listing, 
and  performance  items  are  all 
acceptable  answers  in  the  answer 
key? 

10.  Does  the  test  item  provide 
opportunities  for  COMMON 
ERRORS  to  be  made? 

11.  Is  the  language  of  the  test  item 
easy  for  students  to  understand? 


12.  Is  the  test  item  different  from 
previous  PRACTICE  and 
EXAMPLES?  (USE-CONCEPT, 
USE-RULE,  or 
USE-PRINCIPLE  only) 


(See  table  in  job  aid.) 


1  =  Test  items  for  TLO  or  all 

parts/LOs 

2  =  No  test  items  for  TLO  and 

for  some  parts/LOs 

3  s  No  test  items  for  TLO  and 

for  most  parts/LOs 

1  =  Items  for  all  parts 

2  =  Items  for  many,  but  not  all  parts 

3  =  Items  for  only  a  few  parts  or 

for  no  parts 

(See  table  in  job  aid.) 


1  =  Same 

2  =  Slightly  different 

3  =  Very  different 

1  =  Exact  match 

2  =  Minor  mismatch 

3  s  Severe  mismatch 

1  s  Exact  match 

2  =  Minor  mismatch 

3  =  Severe  mismatch 

I  =  Only  one  answer  is  correct 
3  s  More  than  one  answer  can  be 
correct 

1  =  All  correct  answers  are  in 
answer  key 

3  =  Some  correct  answers  are  not 
in  answer  key 

1  =  Yes 
3  =  No 


1  =  Easy 

2  =  Somewhat  difficult 

3  s  Very  difficult 

1  =  Different 

2  =  Presented  before,  USE-UNAIDED 

3  =  Presented  before,  USE-AIDED 


13.  Is  the  answer  to  the  test  item 
given  away  by  other  item(s)? 


14.  Is  the  answer  to  the  test  item 
dependent  on  answering  previous 
item(s)  correctly? 


15.  Are  sketches  and  diagrams  used 
in  the  test  item  easy  to 
understand? 

16.  Is  the  test  item  tricky  or 
misleading? 


17.  Is  the  test  item  well  constructed? 
(See  job  aid  for  criteria  list  for 
the  test  format  used.) 

18.  When  performance  steps  are 
scored,  does  the  instructor  use 
a  checklist? 


19.  Is  each  correct  answer  position 
used  about  the  same  number  of 
times?  (true-false,  multiple 
choice,  or  matching  items  only) 

20.  Are  specific  patterns  of  correct 
answer  positions  repeated  across 
test  items  or  are  single  positions 
repeated  in  blocks?  (true-false, 
multiple  choice,  or  matching 
items  only) 

21.  Are  test  administration 
directions  complete? 


22.  Do  instructors  follow  the 
directions  when  administering 
the  test? 


23.  Are  adequate  test  instructions 
provided  to  the  student? 


1  =  Answer  not  given  away 

2  =  Other  items  give  clues 

3  =  Answer  can  be  found  in  other 

item(s) 

1  =  Answer  not  dependent  on 
other  items 

3  =  Previous  items  must  be  correctly 
answered 

1  =  Easy  to  understand 

2  =  Somewhat  confusing 

3  =  Very  confusing 

1  =  Not  misleading 

2  =  Somewhat  misleading 

3  *  Very  misleading 

1  =  Meets  all  criteria 

2  =  Deficient  on  noncritical  criteria 

3  =  Deficient  on  critical  criteria 

1  =  Fills  in  completely 

2  s  Uses  as  a  reference  or  fills 

in  partially 

3  =  Does  not  use 

1  =  Yes 
3  =  No 


1  =  No  patterns  easily  seen 
3  =  Patterns  can  be  seen 


1  =  Directions  are  complete 

2  s  Directions  provided,  but 

incomplete  or  unclear 

3  =  Directions  are  not  provided 

1  =  Yes 

2  =  Some  variations  from  directions 

3  =  Significant  variations  from 

directions 

1  =  Yes 

2  s  Instructions  provided,  but 

unclear 

3  =  No  instructions  provided 


24.  Does  the  FINAL  TEST  integrate 
tasks  as  they  are  integrated  in 
the  "real  world?" 

25.  Are  tasks  and  task  steps  tested 
in  the  same  sequence  as  they  are 
performed  in  the  "real  world?" 

26.  In  the  test  free  of  external  cues 
or  help? 


27.  Are  motivational  techniques 
employed? 

28.  Is  the  trainee  attitude  positive? 


29.  Are  course  ENTRY  SKILLS 
reviewed? 


30.  Is  mastery  of  prerequisite  skills 
verified  prior  to  new  instructions? 

31.  Are  OBJECTIVES  presented  to 
the  student? 

32.  Are  the  basic  PRESENTATION 
COMPONENTS  present? 

33.  Are  STATEMENTS  complete? 


34.  Are  STATEMENTS  for  CONCEPTS, 
PROCEDURES,  or  RULES 
adequate?  (See  job  aid  criteria.) 

35.  Does  STATEME^  HELP  provide 
sufficient  explanation? 


36.  Does  training  include  instruction 
on  the  use  of  required  job 
performance  aids? 


1  =  Yes 

2  =  Partially 

3  =  No,  tasks  are  tested  separately 

1  =  Yes 

2  =  Slightly  out  of  sequence 

3  =  Very  different  sequence 

1  =  Yes 

2  =  Hints  given 

3  =  Answers  are  given  away 

1  =  Yes 

3  =  No 

1  =  Positive 

2  =  Indifferent 

3  =  Hostile  or  frustrated 

1  =  Review  with  practice 

2  =  Review  with  no  practice 

3  =  No  review 

1  =  Yes 

3  =  No 


1  =  Yes 
3  *  No 

(See  guidance  and  tables  in 
handbook.) 

1  =  STATEMENT  complete 

2  =  Few  parts  missing 

3  =  Many  parts  missing 

1  =  Completely  adequate 
3  =  Some  or  ail  features  omitted 


1  =  Help  provides  sufficient 

explanation 

2  =  Help  gives  insufficient 

explanation 

3  =  Help  is  confusing 

1  =  Yes 
3  =  No 


37.  Are  EXAMPLES  and 

NON-EXAMPLES  adequate? 


1  =  Yes 
3  s  No 


38.  Is  EXAMPLE  HELP  adequate? 


1  =  Help  provides  sufficient 

explanation 

2  =  Help  gives  insufficient 

explanation 


3  =  Help  is  confusing 

39.  Are  EXAMPLES  sequences  from 

1  =  Yes 

easy  to  hard?  (CONCEPTS  only) 

3  =  No 

40-  Are  there  enough  EXAMPLES? 

1  =  Yes 

(See  job  aid  for  criteria.) 

3  =  No 

41.  Are  NON-EXAMPLES  included? 

1  =  Yes 

(CONCEPTS  only) 

3  =  No 

42.  Do  DEMONSTRATIONS  show 

1  =  Yes 

how  to  correct/avoid  common 

3  =  No 

errors? 

43.  Are  steps  in  a  DEMONSTRATION 

1  =  Yes 

the  appropriate  size?  (See  job 

2  =  Step  size  is  too  small 

aid.) 

3  =  Step  size  is  too  large 

44.  Are  tasks  and  task  steps 

1  =  Yes 

DEMONSTRATED  in  the  same 

2  =  Slightly  out  of  sequence 

sequences  as  they  are  performed 

3  =  Very  different  sequence 

in  the  "real  world?" 

45.  Are  memory  aids  used? 

1  =  Used 

(PRACTICE  REMEMBERING 

3  =  Not  Used 

only) 

46.  Does  each  PRACTICE 

1  =  Same 

REMEMBERING  item  have  the  same 

2  =  Same  content,  different 

content  and  format  as  the  test 

format 

item? 

3  =  Different  content 

47.  Are  PRACTICE  USING  items 

1  =  Yes 

sequenced  from  easy  to  hard? 

3  =  No 

48.  Do  PRACTICE  USING  items 

1  =  Yes 

provide  opportunities  for 

3  =  No 

COMMON  ERRORS  to  be  made? 

49.  Are  PRACTICE  items  different 

1  =  Different 

from  EXAMPLES?  (USE-CONCEPT, 

3  =  Presented  before 

USE-RULE,  or  USE-PRINCIPLE 

only) 

50.  Does  PRACTICE  USING  integrate 

1  =  Yes 

tasks  as  they  are  integrated  in  the 

2  =  Partially 

"real  world?" 

3  =  No,  tasks  are  practiced 

separately 


51.  Are  job  performance  aids 
(JPAs)  usable?  (See  criteria 
in  job  aid.) 

52.  Do  all  students  use  the  job 
performance  aid  (JPA)? 


53.  Does  the  TASK  LEVEL  of  the 
PRACTICE  item  match  that  of  the 
test  item(s)? 

54.  Does  the  CONTENT  TYPE  of 

the  PRACTICE  item  match  that  of 
the  test  item(s)? 

55.  Does  the  FORMAT  of  the 
PRACTICE  item  match  that  of 
the  test  item(s)? 

56.  Do  the  CONDITIONS  of  each 
FINAL  PRACTICE  item  match 
those  of  the  test  item(s)? 

57.  Do  the  STANDARD  of  each 
FINAL  PRACTICE  item  match 
those  of  the  test  item(s)? 

58.  Is  FINAL  PRACTICE  free  of 
external  cues  or  help? 


59.  Are  there  PRACTICE  items  for 
each  TLO  or  all  of  its  critical 
parts/LOs? 


60.  Is  there  a  PRACTICE  items  for 
each  critical  part  of  each  LO? 
(See  job  aid  for  critical  parts.) 


1  =  Easy  to  use 

2  =  Hard  to  use 

3  =  Unusable 

1  =  Yes 

2  =  Up  to  20%  do  not  use  3PA 

3  =  More  than  20%  do  not  use 

3PA 

1  =  Yes 

(2  and  3  —  See  table  in  job  aid.) 


1  -  Yes 
3  =  No 


1  =  Yes 

(2  and  3  —  See  table  in  job  aid.) 


1  =  Yes 

2  =  Slightly  different 

3  =  Very  different 

1  =  Yes 

2  =  Slightly  different 

3  *  Very  different 

1  =  Yes 

2  =  Hints  given 

3  =  Answers  are  given  away 

1  =  PRACTICE  items  for  the 

TLO  or  all  parts /LOs 

2  =  No  PRACTICE  items  for  the 

TLO  and  for  some  parts/LOs 

3  =  No  PRACTICE  items  for  the 

TLO  and  for  most  parts/LOs 

1  =  PRACTICE  for  all  parts 

2  =  PRACTICE  for  many,  but  not 

sill  parts 

3  =  PRACTICE  for  only  a  few 

parts  or  for  no  parts 

1  =  Yes 

2  =  Up  to  20%  of  students  do  not 

PRACTICE 

3  =  More  than  20%  of  the  students 

do  not  PRACTICE 


61.  Do  all  students  PRACTICE? 


62.  Do  all  students  meet  the 

required  STANDARDS  in  FINAL 
PRACTICE? 


63.  Is  FEEDBACK  provided  for 
PRACTICE? 


60.  Is  FEEDBACK  HELP  adequate? 


65.  Is  TEAM  PRACTICE  provided? 


66.  Are  TEAM  PRACTICE 
CONDITIONS  the  same  as  (or  as 
close  as  possible  to)  those  of 
the  real  task? 

67.  Is  TEAM  PRACTICE  FEEDBACK 
provided? 


68.  Is  FEEDBACK  HELP  for  TEAM 
PRACTICE  adequate? 


69.  Are  all  PRESENTATION 
COMPONENTS  separated  and 
identified? 

70.  Is  the  technical  quality  of  written 
or  spoken  material  adequate? 

(See  job  aid  for  criteria.  Make 
notes  on  specific  problems.) 

71.  Is  the  wording  of  written  or 
spoken  material  easy  for  the 
students  to  understand? 


72.  Is  the  instructor's  presentation 
or  the  narration  easy  to  listen  to? 


1  =  Yes 

2  =  Up  to  20%  of  students  do  not 

3  =  More  than  20%  of  students 

do  not 

1  =  FEEDBACK  HELP  is  given 

2  =  Correct  answer  only  is  given 

3  s  No  feedback  is  given 

1  =  Help  gives  enough 

explanation 

2  =  Help  gives  insufficient 

explanation 

3  =  Help  is  confusing 

1  =  Yes 
3  =  No 

1  =  Yes 

2  =  Slightly  different 

3  =  Very  different 


1  =  FEEDBACK  HELP  is  given 

2  =  Success/Failure  feedback 

only  is  given 

3  =  No  feedback  is  given 

1  s  Help  gives  enough  explanation 

2  =  Help  gives  insufficient 

explanation 

3  =  Help  is  confusing 

1  =  Yes 

2  =  Some  are  not 

3  =  Most  or  all  are  not 

1  =  Most  criteria  met 

2  =  Several  criteria  not  met 

3  =  Few  criteria  met 


1  =  Yes,  few  hard  words  and 

long  sentences 

2  =  Some  hard  words  and  long 

sentences 

3  =  Many  hard  words  and  long 

sentences 

1  =  Yes 

2  =  Dull  and  monotonous 

3  =  Hard  to  listen  to 


X 


( 


73.  Is  the  instructor's  presentation 
or  the  narration  supported  by 
visuals? 

74.  Are  visuals  easily  understood? 


75.  Are  the  OBJECTIVES  (TLOs  and 
LOs)  within  each  LESSON 
sequenced  properly? 

(Prerequisites  taught  first.) 

76.  Are  the  LESSONS  sequenced 
properly  within  the  course? 

77.  Are  the  media  appropriate  for  the 
objectives?  (See  table  in  job  aid.) 


73.  Can  the  media  used  provide  all 
necesary  stimuli? 

79.  Are  the  course  administration 
directions  complete? 


80.  Do  course  administration 
directions  make  realistic  demands 
of  students  and  instructors? 

81.  Is  the  instructor /trainee  ratio 
such  that  all  students  can  see, 
hear,  and  receive  feedback? 


82.  Does  the  instructor  follow  the 
methods  in  the  Instructor  Guide? 


83.  Does  the  instructor  teach  all  of 
the  content  in  the  LESSON 
materials? 


84.  Did  the  instructor  limit  his 
teaching  to  the  content  in  the 
LESSON  materials? 


1  =  Completely 

2  =  Partially 

3  =  Not  at  all 

1  =  Yes 

2  =  Understandable  with  effort 

3  =  Very  hard  to  understand 

1  =  Yes 
3  =  No 


1  =  Yes 
3  =  No 

1  =  Yes 

3  a  No  (note  key  words, 
underlined  in  table,  on 
worksheet.) 

1  =  Yes 
3  =  No 

1  =  Yes 

2  =  Partially  incomplete 

3  s  Incomplete  or  non-existent 

l  =  All  demands  are  realistic 
3  =  Some  demands  are  unrealistic 
(Note  what  they  are.) 

1  =  Yes 

2  =  A  few  students  cannot  see, 

hear,  and  receive  feedback 

3  =  Many  students  cannot  see, 

hear,  and  receive  feedback 

1  =  Yes 

2  a  Follows  to  some  extent 

3  a  Follows  very  little  or 

not  at  all 

1  =  Yes 

2  a  Much  of  the  content 

3  =  Very  little  of  the  content 

(If  2  or  3,  note  what  was 
left  out.) 

1  =  Yes 

3  a  No  (Please  note  what  other 
things  he  taught.) 


85.  Is  there  enough  space  for  all  1  =  Yes 

of  the  trainees?  2  =  A  little  crowded 

3  =  Very  cramped  or  some 
students  can't  fit  in  the 
space  at  all 

1  =  Yes 

2  =  Distractions  are  annoying 

3  =  Distractions  seriously 
interfere  with  the 
instruction 

1  =  Yes 

2  =  Students  have  trouble 
reading  or  seeing  displays 
3c  equipment 

3  =  Students  cannot  read  or  see 
displays  3c  equipment 

1  =  Yes 

2  =  Temperature  makes  students 
uncomfortable 

3  =  Temperature  seriously 
interferes  with  learning 

89.  Is  the  instructor's  attitude  1  =  Yes 

positive?  3  =  No 

90.  Are  frequent  breaks  provided?  1  s  Yes 

(5-10  minute  breaks  every  hour)  2  =  Breaks  too  short  or 

infrequent 

3  =  Breaks  not  provided 

91.  Is  the  speed  of  presentation  1  s  Yes 

appropriate?  2  =  Too  slow 

3  *  Too  fast 

92.  Was  the  allotted  training  time  1  =  Appropriate  length 

too  long  or  too  short?  2  =  Too  long 

3  =  Too  short 

1  =  yes 

2  =  Minor  malfunctions,  little 
change  from  intended  task 
performance 

3  a  Major  malfunctions,  substantial 
change  from  intended  task 
performance 

94.  Is  there  anything  else  unusual  about  the  LESSON  materials,  or  do  any  other 
critical  incidents  occur  during  training  that  would  interfere  with  learning? 
(Describe  each  one  below.  Rating  =  3) 


93.  Does  the  training  device/ 
equipment  used  in  training 
function  properly? 


86.  Is  instruction  free  of 
distractions? 


87.  Is  the  lighting  appropriate  for 
the  training  situation? 


88.  Is  the  temperature  appropriate 
for  the  training  situation? 


